Aggregation is thought to enhance an animal’s security through effective predator detection and the dilution of risk. A decline in individual vigilance as group size increases is commonly reported in the literature and called the group size effect. However, to date, most of the research has only been directed toward examining whether this effect occurs at the population level. Few studies have explored the specific contributions of predator detection and risk dilution and the basis of individual differences in the use of vigilance tactics. We tested whether male and female (non-reproductive or with young) eastern grey kangaroos (Macropus giganteus) adopted different vigilance tactics when in mixed-sex groups and varied in their reliance on predator detection and/or risk dilution as group size changed. This species exhibits pronounced sexual dimorphism with females being much smaller than males, making them differentially vulnerable toward predators. We combined field observations with vigilance models describing the effects of detection and dilution on scanning rates as group size increased. We found that females with and without juveniles relied on predator detection and risk dilution, but the latter adjusted their vigilance to the proportion of females with juveniles within their group. Two models appeared to equally support the data for males suggesting that males, similarly to females, relied on predator detection and risk dilution but may also have adjusted their vigilance according to the proportion of mothers within their group. Differential vulnerability may cause sex differences in vigilance tactic use in this species. The presence of males within a group that do not, or only partially, contribute to predator detection and are less at risk may cause additional security costs to females. Our results call for reexamination of the classical view of the safety advantages of grouping to provide a more detailed functional interpretation of gregariousness.